Information processing device, information processing method, and program

ABSTRACT

There is provided an information processing device that enables recognition of a posture relationship between an imaging unit and a detection unit in a more favorable mode. The information processing device includes an acquisition unit configured to acquire an image captured by an imaging unit and a detection result of a gravitational acceleration by a detection unit supported by the imaging unit, for each of a plurality of viewpoints different from one another, an image processing unit configured to extract a plurality of line segments extending in a gravity direction from the image and specify an intersection of straight lines obtained by respectively extending the plurality of line segments on the basis of a subject captured in the image for the each viewpoint, and a calculation unit configured to calculate a relative posture relationship between the imaging unit and the detection unit on the basis of the detection result of the gravitational acceleration acquired for each of the plurality of viewpoints and the intersection specified for the each of the plurality of viewpoints.

CROSS REFERENCE TO PRIOR APPLICATION

This application is a National Stage Patent Application of PCTInternational Patent Application No. PCT/JP2019/007418 (filed on Feb.27, 2019) under 35 U.S.C. § 371, which claims priority to JapanesePatent Application No. 2018-089793 (filed on May 8, 2018), which are allhereby incorporated by reference in their entirety.

TECHNICAL FIELD

The present disclosure relates to an information processing device, aninformation processing method, and a program.

BACKGROUND ART

With the development of image processing technologies, variousrecognition technologies using a captured image have been proposed, suchas analyzing an image captured by an imaging unit like a digital camera(in other words, an image sensor), thereby recognizing a subjectcaptured in the image and recognizing an environment around the imagingunit. In addition, in recent years, a more advanced recognitiontechnology using an analysis result of a captured image and a detectionresult of a detection unit such as an acceleration sensor or an angularvelocity sensor (for example, an inertial measurement unit (IMU)) hasbeen also proposed. Such recognition technologies enable implementationof self-position estimation of a device in which the imaging unit andthe detection unit are supported, for example. Therefore, therecognition technologies are also applied to, for example, augmentedreality (AR) technologies, virtual reality (VR) technologies, androbotics.

CITATION LIST Patent Document

Patent Document 1: Japanese Patent Application Laid-Open No. 2015-117943

Patent Document 2: Japanese Patent No. 3728900

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

In the above-described recognition technologies, there are many caseswhere the imaging unit and various sensors used for recognition arecalibrated to maintain recognition accuracy. For example, PatentDocument 1 discloses an example of a technology regarding calibration ofa sensor that can be used for implementing the above-describedrecognition technologies. Furthermore, Patent Document 2 discloses anexample of a technology regarding calibration of an imaging unit thatcan be used for implementing the above-described recognitiontechnologies.

In particular, in the recognition technology using an analysis result ofa captured image by an imaging unit and a detection result by adetection unit, accurate recognition of a relative posture of one unitto the other is important between the imaging unit and the detectionunit. In other words, to implement the above-described recognitiontechnology in a more favorable mode, introduction of a technology (forexample, a technology regarding calibration) that enables a decrease ininfluence of variation (error) of the relative posture of one unit tothe other unit between the imaging unit and the detection unit, thevariation (error) being caused when the imaging unit and the detectionunit are attached, is required.

Therefore, the present disclosure proposes a technology for enablingrecognition of a posture relationship between an imaging unit and adetection unit in a more favorable mode.

Solutions to Problems

According to the present disclosure, provided is an informationprocessing device including: an acquisition unit configured to acquirean image captured by an imaging unit and a detection result of agravitational acceleration by a detection unit supported by the imagingunit, for each of a plurality of viewpoints different from one another;an image processing unit configured to extract a plurality of linesegments extending in a gravity direction from the image and specify anintersection of straight lines obtained by respectively extending theplurality of line segments on the basis of a subject captured in theimage for the each viewpoint; and a calculation unit configured tocalculate a relative posture relationship between the imaging unit andthe detection unit on the basis of the detection result of thegravitational acceleration acquired for each of the plurality ofviewpoints and the intersection specified for each of the plurality ofviewpoints.

Furthermore, according to the present disclosure, provided is aninformation processing method, by a computer, including: acquiring animage captured by an imaging unit and a detection result of agravitational acceleration by a detection unit supported by the imagingunit, for each of a plurality of viewpoints different from one another;extracting a plurality of line segments extending in a gravity directionfrom the image and specify an intersection of straight lines obtained byrespectively extending the plurality of line segments on the basis of asubject captured in the image for the each viewpoint; and calculating arelative posture relationship between the imaging unit and the detectionunit on the basis of the detection result of the gravitationalacceleration acquired for each of the plurality of viewpoints and theintersection specified for each of the plurality of viewpoints.

Furthermore, according to the present disclosure, provided is a programfor causing a computer to execute: acquiring an image captured by animaging unit and a detection result of a gravitational acceleration by adetection unit supported by the imaging unit, for each of a plurality ofviewpoints different from one another; extracting a plurality of linesegments extending in a gravity direction from the image and specify anintersection of straight lines obtained by respectively extending theplurality of line segments on the basis of a subject captured in theimage for the each viewpoint; and calculating a relative posturerelationship between the imaging unit and the detection unit on thebasis of the detection result of the gravitational acceleration acquiredfor each of the plurality of viewpoints and the intersection specifiedfor each of the plurality of viewpoints.

Effects of the Invention

As described above, according to the present disclosure, there isprovided a technology for enabling recognition of a posture relationshipbetween an imaging unit and a detection unit in a more favorable manner.

Note that the above-described effect is not necessarily limited, and anyof effects described in the present specification or other effects thatcan be grasped from the present specification may be exerted in additionto or in place of the above-described effect.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an explanatory diagram for describing an outline of an exampleof a system configuration of an information processing system accordingto an embodiment of the present disclosure.

FIG. 2 is a block diagram illustrating an example of a functionalconfiguration of the information processing system according to theembodiment.

FIG. 3 is an explanatory diagram for describing an outline of aprinciple of calibration according to the embodiment.

FIG. 4 is an explanatory diagram for describing an outline of thecalibration according to the embodiment.

FIG. 5 is an explanatory diagram for describing an outline of thecalibration according to the embodiment.

FIG. 6 is a flowchart illustrating an example of a flow of processing ofthe calibration according to the embodiment.

FIG. 7 is an explanatory diagram for describing an example of acalibration method according to Modification 1.

FIG. 8 is an explanatory diagram for describing another example of thecalibration method according to Modification 1.

FIG. 9 is an explanatory diagram for describing an outline of aninformation processing system according to Modification 3.

FIG. 10 is an explanatory diagram for describing an outline of theinformation processing system according to Modification 3.

FIG. 11 is a functional block diagram illustrating an example of ahardware configuration of an information processing device configuringan information processing system according to an embodiment of thepresent disclosure.

FIG. 12 is an explanatory diagram for describing an application of theembodiment of the present disclosure.

MODE FOR CARRYING OUT THE INVENTION

A favorable embodiment of the present disclosure will be described indetail with reference to the appended drawings. Note that, in thepresent specification and drawings, redundant description ofconfiguration elements having substantially the same functionalconfiguration is omitted by providing the same sign.

Note that the description will be given in the following order.

1. System Configuration

2. Functional Configuration

3. Calibration Method

3.1. Calibration Principle

3.2. Calibration Processing Flow

4. Modification

4.1. Modification 1: Example of Calibration Method

4.2. Modification 2: Example of Calculation Method of CoordinateConversion Function

4.3. Modification 3: UI Example

5. Hardware Configuration

6. Application

7. Conclusion

1. SYSTEM CONFIGURATION

First, an example of a schematic system configuration of an informationprocessing system according to an embodiment of the present disclosurewill be described with reference to FIG. 1. FIG. 1 is an explanatorydiagram for describing an outline of an example of a systemconfiguration of an information processing system according to anembodiment of the present disclosure.

As illustrated in FIG. 1, an information processing system 1 accordingto the present embodiment includes an information acquisition device 130and an information processing device 100. The information acquisitiondevice 130 and the information processing device 100 are configured tobe able to transmit and receive information to and from each other via apredetermined network. Note that the type of the network connecting theinformation acquisition device 130 and the information processing device100 is not especially limited. As a specific example, the network may beconfigured by a so-called wireless network such as a network based on aWi-Fi (registered trademark) standard. Furthermore, as another example,the network may be configured by the Internet, a dedicated line, a localarea network (LAN), a wide area network (WAN), or the like. Furthermore,the network may include a plurality of networks, and at least a part ofthe networks may be configured as a wired network.

The information acquisition device 130 schematically illustrates adevice for acquiring information used for implementing variousrecognition technologies (in other words, information used forrecognizing various states). As a specific example, in a case ofassuming a situation of recognizing various states for self-positionestimation, the information acquisition device 130 corresponds to amoving body to be estimated for a self-position or a device thatcollects various types of information for implementing the self-positionestimation supported by the moving body.

As illustrated in FIG. 1, the information acquisition device 130includes an imaging unit 131 for capturing an image of a surroundingenvironment, and a detection unit 133 (for example, an IMU) fordetecting changes in the position and posture of the informationacquisition device 130, such as acceleration and angular velocity. Theimaging unit 131 and the detection unit 133 are supported such that arelative relationship of at least the postures satisfies a predeterminedrelationship between the respective positions and postures. As aspecific example, the imaging unit 131 and the detection unit 133 arefavorably supported such that the relative posture relationship is fixedat least at the time of acquiring information for calibration to bedescribed below. As a more specific example, the imaging unit 131 andthe detection unit 133 are favorably integrally supported. Furthermore,the imaging unit 131 and the detection unit 133 may be configured suchthat one unit is attachable to and detachable from the other unit.

The information processing device 100 calibrates each configuration suchas the imaging unit 131 and the detection unit 133 supported by theinformation acquisition device 130. In particular, the informationprocessing device 100 according to the present embodiment calibratesvariation in the relative posture of one unit to the other unit betweenthe imaging unit 131 and the detection unit 133, which may occur foreach information acquisition device 130.

Specifically, there are some cases where installation of the imagingunit 131 and the detection unit 133 at the same position is practicallydifficult. In this case, a local coordinate system of the imaging unit131 (that is, a coordinate system indicating relative position anddirection with respect to the imaging unit 131 as a base point) and alocal coordinate system of the detection unit 133 (that is, a coordinatesystem indicating relative position and direction with respect to thedetection unit 133 as a base point) do not necessarily match. In such asituation, the local coordinate system of the imaging unit 131 and thelocal coordinate system of the detection unit 133 may be made tocorrespond to each other. More specifically, conversion from onecoordinate system into the other coordinate system may be performedbetween the local coordinate system of the imaging unit 131 and thelocal coordinate system of the detection unit 133, by using apredetermined conversion function. Meanwhile, there are some casesvariation occurs in the relative relationship between the posture of theimaging unit 131 and the posture of the detection unit 133, for eachinformation acquisition device 130, according to errors relating toattachment of the imaging unit 131 and the detection unit 133 or thelike. Such variation may be one of factors of reduction in recognitionaccuracy in recognition processing using an imaging result of an imageby the imaging unit 131 and a detection result by the detection unit133, for example. In view of such a situation, the informationprocessing device 100 according to the present embodiment calibrates thevariation in the relative posture of one unit to the other unit betweenthe imaging unit 131 and the detection unit 133, which may occur foreach information acquisition device 130.

Specifically, the information processing device 100 acquires an imagecaptured by the imaging unit 131 from a viewpoint and a detection resultof gravitational acceleration by the detection unit 133 at the viewpointfrom the information acquisition device 130, for each of a plurality ofviewpoints different from one another. Note that, in the presentdisclosure, the “viewpoint” indicates a position or a posture of theinformation acquisition device 130 (and thus the imaging unit 131). As amore specific example, the “viewpoint” in the present disclosurecorresponds to, in a case where an optical axis of the imaging unit 131is viewed as a line-of-sight, the position where the imaging unit 131serving as a base point of the line-of-sight is held or the posture ofthe imaging unit 131. At this time, the position of each viewpoint and asubject to be captured by the imaging unit 131 are set such that aplurality of line segments extending in a gravity direction is extractedfrom the image captured by the imaging unit 131. For example, in theexample illustrated in FIG. 1, thread-like objects M101 a and M101 bwith fixed one ends are the subjects, and the imaging unit 131 capturesimages of the objects M101 a and M101 b from a plurality of viewpointsdifferent from one another. Note that details of a calibration methodwill be described below.

An example of a schematic system configuration of an informationprocessing system according to an embodiment of the present disclosurehas been described with reference to FIG. 1. Note that theabove-described system configuration is a mere example, and does notnecessarily limit the system configuration of the information processingsystem 1 according to the present embodiment. As a specific example, theinformation acquisition device 130 and the information processing device100 may be integrally configured. Furthermore, the functions of theinformation processing device 100 may be implemented by a plurality ofdevices that operates in cooperation with each other.

2. FUNCTIONAL CONFIGURATION

Next, an example of a functional configuration of the informationprocessing system according to the embodiment of the present disclosurewill be described with reference to FIG. 2. FIG. 2 is a block diagramillustrating an example of a functional configuration of the informationprocessing system according to the present embodiment.

As illustrated in FIG. 2, the information processing system 1 accordingto the present embodiment includes an information acquisition device130, an information processing device 100, and a storage unit 151.Furthermore, the information processing system 1 according to thepresent embodiment may include an input unit 153 and an output unit 155.Note that the information acquisition device 130 corresponds to theinformation acquisition device 130 illustrated in FIG. 1. That is, animaging unit 131 and a detection unit 133 correspond to the imaging unit131 and the detection unit 133 illustrated in FIG. 1. Therefore,detailed description of the information acquisition device 130, and theimaging unit 131 and the detection unit 133 included in the informationacquisition device 130 will be omitted.

The information processing device 100 corresponds to the informationprocessing device 100 illustrated in FIG. 1. As illustrated in FIG. 2,the information processing device 100 includes an image processing unit101 and a calculation unit 103. Furthermore, the information processingdevice 100 may also include an input/output control unit 105.

The image processing unit 101 acquires an image captured by the imagingunit 131 for each of a plurality of viewpoints, and applies analysisprocessing (image analysis) to the image captured for the eachviewpoint, thereby extracting a plurality of line segments extending inthe gravity direction from the image on the basis of a subject capturedin the image. Furthermore, the image processing unit 101 specifies anintersection of straight lines obtained by respectively extending theplurality of line segments in the image on the basis of an extractionresult of the plurality of line segments from the image. Then, the imageprocessing unit 101 outputs, for each of the plurality of viewpoints,information indicating the intersection of the straight lines obtainedby respectively extending the plurality of line segments extracted fromthe image corresponding to the viewpoint to the calculation unit 103.

Furthermore, the image processing unit 101 may cause a user to specify asubject from which the line segment is to be extracted and a linesegment to be used for specifying the intersection. In this case, theimage processing unit 101 may present the image captured by the imagingunit 131 to the user by causing a predetermined output unit (forexample, the output unit 155 to be described below) to output the imagevia the input/output control unit 105 to be described below.Furthermore, the image processing unit 101 may acquire information forthe image presented to the user via the input/output control unit 105,the information according to specification by the user via apredetermined input unit (for example, the input unit 153 to bedescribed below). Then, the image processing unit 101 may determine thesubject from which the line segment is to be extracted and the linesegment to be used for specifying the intersection according to thespecification by the user.

The calculation unit 103 acquires information according to theacceleration detection result by the detection unit 133 to specify thedirection of gravitational acceleration (in other words, a gravitationalacceleration vector) in the local coordinate system of the detectionunit 133, for each of the plurality of viewpoints. As a specificexample, the acceleration detected by the detection unit 133 in a statewhere the detection unit 133 (and thus the information acquisitiondevice 130) is stationary can correspond to the gravitationalacceleration. Therefore, for example, the calculation unit 103recognizes information indicating the acceleration detection result bythe detection unit 133 acquired in the state where the detection unit133 is stationary, as the information indicating the gravitationalacceleration detection result by the detection unit 133. Of course, themethod is not particularly limited as long as the direction of thegravitational acceleration in the local coordinate system of thedetection unit 133 can be specified according to the detection result ofthe acceleration or the angular velocity by the detection unit 133.Furthermore, at this time, the calculation unit 103 may specify thedirection of the gravitational acceleration in consideration ofcalibration data (for example, a bias and a scale factor) of thedetection unit 133, which has been acquired in advance. The calibrationdata of the detection unit 133 may be stored in advance in a storagearea that the calculation unit 103 can refer to (for example, thestorage unit 151 to be described below). Note that, in the followingdescription, the gravitational acceleration vector (that is, thegravitational acceleration direction) in the local coordinate system ofthe detection unit 133, which is specified as described above, may bereferred to as a “gravitational acceleration vector a” for convenience.Furthermore, the local coordinate system of the detection unit 133corresponds to an example of a “second coordinate system”.

Furthermore, the calculation unit 103 acquires, for each of theplurality of viewpoints at which the acceleration detection result hasbeen acquired by the detection unit 133, the information indicating theintersection of the straight lines obtained by respectively extendingthe plurality of line segments extracted from the image corresponding tothe viewpoint from the image processing unit 101. The calculation unit103 calculates, for each of the plurality of viewpoints, a vectorindicating the gravity direction in the local coordinate system of theimaging unit 131, the vector corresponding to the viewpoint, on thebasis of the information indicating the intersection of the straightlines obtained by respectively extending the plurality of line segments.Furthermore, at this time, the calculation unit 103 may calculate thevector in the gravity direction in consideration of calibration data(for example, lens distortion and a calibration result of internalparameters) of the imaging unit 131, which has been acquired in advance.The calibration data of the imaging unit 131 may be stored in advance ina storage area that the calculation unit 103 can refer to (for example,the storage unit 151 to be described below). Note that, in the followingdescription, the vector in the gravity direction in the local coordinatesystem of the imaging unit 131, which is specified as described above,may be referred to as a “gravity direction vector c” for convenience.Furthermore, the local coordinate system of the imaging unit 131corresponds to an example of a “first coordinate system”.

Then, the calculation unit 103 calculates a relative posturerelationship between the imaging unit 131 and the detection unit 133 onthe basis of the gravitational acceleration vector a and the gravitydirection vector c at each of the plurality of viewpoints. Specifically,the calculation unit 103 calculates a function R for converting one unitinto another unit between the local coordinate system of the imagingunit 131 and the local coordinate system of the detection unit 133 onthe basis of the gravitational acceleration vector a and the gravitydirection vector c for each viewpoint. Then, the calculation unit 103causes a predetermined storage area (for example, the storage unit 151to be described below) to store information indicating a calculationresult of the function R. Note that the function R calculated at thistime is calculated according to actual installation positions of theimaging unit 131 and the detection unit 133. Therefore, by using thefunction R for coordinate conversion between the imaging unit 131 andthe detection unit 133, conversion considering errors relating toattachment of the imaging unit 131 and the detection unit 133 (that is,conversion in which the error is calibrated) can be performed. Note thatdetails of the method of calculating the function R (that is,calibration) will be described below.

Furthermore, the calculation unit 103 may cause a predetermined outputunit to output, via the input/output control unit 105, information forguiding movement of the information acquisition device 130 to theposition of a new viewpoint so as to acquire information of the newviewpoint according to an acquisition situation of information of eachviewpoint. The user guides the information acquisition device 130 tochange the position and posture by the control, so that acquisition ofthe information regarding the gravitational acceleration vector a andcalculation of the information regarding the gravity direction vector cbecome possible. In other words, in a case where the number of samplesis insufficient as information for calculating the function R (theabove-described information acquired for each viewpoint), the user canbe guided to enable acquisition of new samples for calculating thefunction R in a more favorable mode. Note that an example of a userinterface (UI) regarding the guidance will be described below asmodifications.

The storage unit 151 is a storage area for temporarily or constantlystoring various data. For example, the storage unit 151 may store datafor the information processing device 100 to execute various functions.As a specific example, the storage unit 151 may store data (for example,a library) for executing various applications, management data formanaging various settings, and the like. Furthermore, the storage unit151 may store the information according to the result of calibrationperformed by the calculation unit 103 (that is, the function R forperforming coordinate conversion between the imaging unit 131 and thedetection unit 133).

The input unit 153 corresponds to an input interface for the user toinput various types of information to the information processing device100. The input unit 153 may include, for example, an input device suchas a button, a lever, and a touch panel. Note that the configuration ofthe input unit 153 is not especially limited as long as the user caninput information to the information processing device 100. As aspecific example, the input unit 153 may include a sound collection unitfor collecting voice of the user and may acquire the voice of the userinput to the sound collection unit as information input by the user.

The output unit 155 is an output interface for presenting various typesof information to the user by the information processing device 100. Theoutput unit 155 may include, for example, a display device that outputsan image such as a still image or a moving image, such as a so-calleddisplay. Note that the configuration of the output unit 155 is notespecially limited as long as the information processing device 100 canpresent various types of information to the user. As a specific example,the output unit 155 may include an acoustic device that outputs sound,such as a speaker. Furthermore, the output unit 155 may include avibrating device that presents information to the user by vibrating in apattern corresponding to the information to be presented, such as aso-called vibrator.

Note that, although not explicitly illustrated in FIG. 2, configurationscorresponding to a portion that acquires the image captured by theimaging unit 131 and a portion that acquires the detection result fromthe detection unit 133, of the information processing device 100,correspond to an example of an “acquisition unit”. That is, the portionthat acquires the image captured by the imaging unit 131, of the imageprocessing unit 101, and a portion that acquires the detection resultfrom the detection unit 133, of the calculation unit 103, can correspondto an example of the “acquisition unit”. In other words, the interfaceof the information processing device 100 for acquiring the various typesof information from the imaging unit 131 and the detection unit 133 cancorrespond to an example of the “acquisition unit”.

An example of a functional configuration of the information processingsystem according to the embodiment of the present disclosure has beendescribed with reference to FIG. 2. Note that the above-describedfunctional configuration is a mere example, and does not necessarilylimit the functional configuration of the information processing system1 according to the present embodiment. As a specific example, theinformation acquisition device 130 and the information processing device100 may be integrally configured, as described above. Furthermore, asanother example, at least part of the configurations of the storage unit151, the input unit 153, and the output unit 155 may be included in theinformation processing device 100. Furthermore, part of theconfigurations of the information processing device 100 may be providedoutside the information processing device 100. Furthermore, thefunctions of the information processing device 100 may be implemented bya plurality of devices that operate in cooperation with each other.

3. CALIBRATION METHOD

Next, hereinafter, an example of a method of calibrating the relativeposture relationship between the imaging unit 131 and the detection unit133, which is executed by the information processing device 100according to the embodiment of the present disclosure, will be describedin detail. Note that, in the following description, the calibrationexecuted by the information processing device 100 will be simplyreferred to as “calibration according to the present embodiment” forconvenience.

<3.1. Calibration Principle>

First, an outline of a principle of calibration according to the presentembodiment will be described with reference to FIG. 3. FIG. 3 is anexplanatory diagram for describing an outline of a principle ofcalibration according to the present embodiment. Note that in FIG. 3,reference numerals 130 a and 130 b schematically illustrate viewpointsdifferent from each other at which the information acquisition device130 is held (in other words, the information processing devices 130 heldat positions and postures different from each other).

(Calculation of Gravity Direction Vector c)

The information processing device 100 according to the presentembodiment extracts a plurality of line segments extending in thegravity direction from the image captured by the imaging unit 131supported by the information acquisition device 130, and specifies theintersection of the line segments obtained by extending the plurality ofline segments, thereby calculating the gravity direction vector c.Therefore, in the calibration according to the present embodiment, it isdesirable to use an object having a portion extending in the gravitydirection as a subject.

For example, in the example illustrated in FIG. 3, the thread-likeobjects M101 a and M101 b having fixed one ends and the other ends towhich a weight is attached are used as subjects. By capturing an imageof the objects M101 a and M101 b as subjects, the objects M101 a andM101 b in a state of being stretched to extend in the gravity directionby the weights are captured in the image.

The position and posture (in other words, the viewpoint) of the imagingunit 131 that captures the image of the objects M101 a and M101 b aredesirably set such that the objects M101 a and M101 b captured assubjects in the image are not parallel to each other. Therefore, forexample, in the example illustrated in FIG. 3, the position and posture(in other words, the viewpoint) of the imaging unit 131 that capturesthe image are desirably set such that an image in which the objects M101a and M101 b are looked down from obliquely above is captured.

For example, FIG. 4 is an explanatory diagram for describing an outlineof the calibration according to the present embodiment, and illustratesan example of an image in which the objects M101 a and M101 billustrated in FIG. 3 are captured to be looked down from obliquelyabove. As illustrated in FIG. 4, in the case where the objects M101 aand M101 b are not parallel to each other in the image, portionsobtained by extending the respective thread-like portions of the objectsM101 a and M101 b intersect at part of a coordinate system of the image(that is, the local coordinate system of the imaging unit 131). Forexample, reference symbol P illustrated in FIG. 4 represents theintersection of the extended portions of the objects M101 a and M101 b.

Note that the objects M101 a and M101 b are held such that thethread-like portions extend in the gravity direction in a real space.That is, the direction in which the thread-like portions of the objectsM101 a and M101 b extend on the image indicates the gravity direction inthe coordinate system of the image. Furthermore, in the case of theexample illustrated in FIG. 4, an intersection P of the portionsobtained by extending the objects M101 a and M101 b captured as subjectsin the coordinate system of the image corresponds to a vanishing point(infinite point) in the gravity direction (vertically downwarddirection).

The information processing device 100 according to the presentembodiment calculates the gravity direction vector c in the localcoordinate system of the imaging unit 131 on the basis of the subjectsin the image captured by the imaging unit 131, by using theabove-described characteristics.

Note that the method of extracting the thread-like portions (that is,the line segments extending in the gravity direction) of the objectsM101 a and M101 b from the image captured by the imaging unit 131 is notparticularly limited. For example, an image of a case where the objectsM101 a and M101 b (especially, the thread-like portions extending in thegravity direction) are present in a capturing range and an image of acase where the objects M101 a and M101 b are not present are capturedunder the same exposure setting without moving the position and postureof the imaging unit 131 (that is, without moving the viewpoint), and theobjects M101 a and M101 b may be extracted as a difference between thetwo images. Furthermore, as another example, the thread-like portions(that is, the line segments extending in the gravity direction) of theobjects M101 a and M101 b may be extracted by extracting portions fit toa straight line from an image captured by two-dimensional Houghconversion. The above-description is mere examples, and the method isnot particularly limited as long as a plurality of line segmentsextending in the gravity direction can be extracted from an image on thebasis of a subject captured in the image.

Furthermore, in a case where the number of line segments extracted froman image for each viewpoint is three or more, all the straight linesobtained by respectively extending the plurality of extracted linesegments may not intersect at one point due to an influence ofobservation errors or the like. In such a case, a point where a sum ofdistances to the respective straight lines becomes the smallest may beregarded as the above-mentioned intersection. Furthermore, theintersection of the straight lines obtained by respectively extendingthe plurality of line segments is not necessarily located in the image.That is, even in a case where the intersection is located outside theimage, coordinates of the intersection in the local coordinate system ofthe imaging unit 131 are specified, whereby the gravity direction vectorc can be calculated on the basis of a specification result of theintersection.

Here, the method of calculating the gravity direction vector c will bespecifically described below. Note that, in the following description,for convenience, an optical axis direction of the imaging unit 131 is az direction, and directions corresponding to a horizontal direction anda vertical direction of the captured image on an imaging plane of theimaging element of the imaging unit 131 are an x direction and a ydirection, respectively, in the local coordinate system of the imagingunit 131. Furthermore, regarding the imaging unit 131, an internalparameter matrix K of the imaging unit 131 is expressed by(Expression 1) below, in a case where focal lengths in the x directionand the y direction are f_(x) and f_(y), respectively, and opticalcenters in the x direction and the y direction are c_(x) and c_(y),respectively.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 1} \right\rbrack & \; \\{K = \begin{bmatrix}f_{x} & 0 & c_{x} \\0 & f_{y} & c_{y} \\0 & 0 & 1\end{bmatrix}} & \left( {{Expression}\mspace{14mu} 1} \right)\end{matrix}$

At this time, when a two-dimensional point (in other words, a point onthe image) in a two-dimensional coordinate system (xy plane) on theimage captured by the imaging unit 131 is (p_(x), p_(y)), thetwo-dimensional point can be converted into a three-dimensional vector(v_(x), v_(y), v_(z)) in the real space on the basis of a calculationexpression illustrated as (Expression 2) below.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 2} \right\rbrack & \; \\{\begin{bmatrix}v_{x} \\v_{y} \\v_{z}\end{bmatrix} = {K^{- 1}\begin{bmatrix}p_{x} \\p_{y} \\1\end{bmatrix}}} & \left( {{Expression}\mspace{14mu} 2} \right)\end{matrix}$

That is, the information processing device 100 converts the coordinatesin the two-dimensional coordinate system on the image captured by theimaging unit 131, of the intersection (for example, the intersection Pillustrated in FIG. 4) of the plurality of line segments extending inthe gravity direction extracted from the captured image by the imagingunit 131, into a three-dimensional vector on the basis of (Expression 1)and (Expression 2). The three-dimensional vector thus obtainedcorresponds to the gravity direction vector c in the real space.

Note that, as described above, the information processing device 100specifies the intersection of the plurality of line segments extractedfrom the image, thereby calculating the gravity direction vector c.Therefore, it is more desirable to set the viewpoint (in other words,the position and posture of the imaging unit 131) such that theintersection can be stably specified. Here, an example of settingconditions of the viewpoint at which the intersection (and thus thegravity direction vector c) can be more stably calculated will bedescribed with reference to FIG. 5. FIG. 5 is an explanatory diagram fordescribing an outline of the calibration according to the presentembodiment, and illustrates an example of setting conditions of theviewpoint at which the gravity direction vector c can be calculated in amore favorable mode.

In FIG. 5, reference numerals M101 a and M101 b schematically representportions (line segments) extending in the gravity direction, of theobjects M101 a and M101 b illustrated in FIGS. 3 and 4. Reference symbold_(near) schematically represents a distance between one of the ends ofthe objects M101 a and M101 b (line segments) along the gravitydirection, the one end closer to the imaging unit 131, and the imagingunit 131. Furthermore, reference symbol d_(far) schematically representsa distance between one of the ends of the objects M101 a and M101 b(line segments) along the gravity direction, the one end farther fromthe imaging unit 131, and the imaging unit 131. At this time, theviewpoint (in other words, the position and posture of the imaging unit131, and thus the position and posture of the information acquisitiondevice 130) is set such that the distances d_(near) and d_(far) satisfythe condition described as (Expression 3) below, so that theintersection (and thus the gravity direction vector c) can be morestably calculated.[Math. 3]d _(far)>2d _(near)  (Expression 3)

(Calculation of Function R)

Next, an example of a method of calculating the function R forrotationally converting one coordinate system into the other coordinatesystem between the local coordinate system of the imaging unit 131 andthe local coordinate system of the detection unit 133 will be described.As described above, the information processing device 100 according tothe present embodiment calculates the function R on the basis of thegravity direction vector c and the gravitational acceleration vector afor each of the plurality of viewpoints. Note that the method ofcalculating the gravity direction vector c is as described above.Furthermore, the gravitational acceleration vector a can be acquired asthe detection result of the acceleration (gravitational acceleration) bythe detection unit 133, as described above. Here, in a case where thefunction R is a function for rotationally converting the localcoordinate system of the detection unit 133 into the local coordinatesystem of the imaging unit 131, the relationship between a gravitydirection vector c_(n) and a gravitational acceleration vector a_(n) ata certain viewpoint n is expressed as the relational expression as(Expression 4) below.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 4} \right\rbrack & \; \\{\frac{c_{n}}{c_{n}} = {R\frac{a_{n}}{a_{n}}}} & \left( {{Expression}\mspace{14mu} 4} \right)\end{matrix}$

The function R can be estimated by repeating the observation (forexample, the observation described with reference to FIGS. 3 and 4) Ntimes (N is an integer of 2 or larger) and searching for the function Rthat satisfies the relationship described in (Expression 4) the best.That is, the function R is calculated on the basis of the calculationexpression described as (Expression 4) below. Note that it is assumedthat influence of errors (for example, a distortion internal parameterof a lens or the like) caused by the imaging unit 131 itself, errors(for example, the bias and scale factor) caused by the detection unit133 itself, and the like can be ignored by the calibration appliedthereto in advance.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 5} \right\rbrack & \; \\{R = {\underset{R}{\arg\;\min}{\sum\limits_{n = {1\mspace{11mu}\ldots\mspace{14mu} N}}\left\{ {\frac{c_{n}}{c_{n}} - {R\frac{a_{n}}{a_{n}}}} \right\}}}} & \left( {{Expression}\mspace{14mu} 5} \right)\end{matrix}$

As described above, the function R is estimated by searching for thefunction R that satisfies the relationship described in (Expression 5)the best. It is desirable to set each viewpoint such that differentresults can be obtained as the gravity direction vector c_(n) and thegravitational acceleration vector an among the plurality of viewpointsfrom the above characteristic. Specifically, it is favorable to set eachviewpoint such that the position of the intersection specified from theimage captured for each viewpoint (that is, the position in the localcoordinate system of the imaging unit 131) is different between at leasttwo or more viewpoints among the plurality of viewpoints.

An outline of the principle of the calibration according to the presentembodiment has been described with reference to FIGS. 3 to 5.

<3.2. Calibration Processing Flow>

Next, an example of a flow of calibration processing according to thepresent embodiment will be described with reference to FIG. 6. FIG. 6 isa flowchart illustrating an example of a flow of processing of thecalibration according to the present embodiment.

As illustrated in FIG. 6, the information processing device 100 (imageprocessing unit 101) according to the present embodiment acquires theimage captured by the imaging unit 131 supported by the informationacquisition device 130 from the information acquisition device 130.Furthermore, the information processing device 100 (calculation unit103) acquires the information according to the detection result of thegravitational acceleration by the detection unit 133 supported by theinformation acquisition device 130 from the information acquisitiondevice 130 (S101).

The information processing device 100 (image processing unit 101)applies the analysis processing (image analysis) to the image capturedfor the each viewpoint, thereby extracting the plurality of linesegments extending in the gravity direction from the image on the basisof a subject captured in the image (S103).

The information processing device 100 (image processing unit 101)specifies the intersection of the straight lines obtained byrespectively extending the plurality of line segments in the image onthe basis of the extraction result of the plurality of line segmentsfrom the image (S105).

The information processing device 100 continues the processing describedin reference wharfs S101 to S105 until the number of samples of theabove-described various types of information acquired for each viewpointbecomes a threshold or larger while changing the position and posture(that is, the viewpoint) of the information acquisition device 130(S107, NO).

Then, in the case where the number of samples of the various types ofinformation acquired for each viewpoint becomes the threshold or larger(S107, YES), the information processing device 100 (calculation unit103) calculates the relative posture relationship between the imagingunit 131 and the detection unit 133 on the basis of the various types ofinformation of the viewpoint (S109).

Specifically, the information processing device 100 (calculation unit103) specifies the gravitational acceleration vector a in the localcoordinate system of the detection unit 133 corresponding to theviewpoint according to the acceleration detection result by thedetection unit 133 acquired for each of the plurality of viewpoints.Furthermore, the information processing device 100 (calculation unit103) calculates, for the each of the plurality of viewpoints, thegravity direction vector c in the local coordinate system of the imagingunit 131 corresponding to the viewpoint on the basis of thespecification result of the intersection of the plurality of linesegments corresponding to the viewpoint. Then, the informationprocessing device 100 (calculation unit 103) calculates the function Rfor converting one coordinate system into another coordinate systembetween the local coordinate system of the imaging unit 131 and thelocal coordinate system of the detection unit 133 on the basis of thegravitational acceleration vector a and the gravity direction vector cfor each of the plurality of viewpoints.

An example of the flow of the calibration processing according to thepresent embodiment has been described with reference to FIG. 6.

4. MODIFICATION

Next, modifications of the information processing system 1 according tothe present embodiment will be described.

<4.1. Modification 1: Example of Calibration Method>

First, as Modification 1, another example of the calibration methodaccording to the present embodiment will be described especiallyfocusing on a method of calculating the gravity direction vector c inthe local coordinate system of the imaging unit 131 on the basis of asubject in the image captured by the imaging unit 131.

As described above, in the calibration according to the presentembodiment, the plurality of line segments is extracted from the imageon the basis of the subject captured in the image, and the intersection(vanishing point) of the straight lines obtained by respectivelyextending the plurality of line segments is specified. Then, the gravityvector c in the local coordinate system of the imaging unit 131 iscalculated on the basis of the specification result of the intersectionand the internal parameter of the imaging unit 131. Therefore, forexample, in the example described with reference to FIGS. 3 to 5, thethread-like objects M101 a and M101 b having fixed one ends and theother ends to which a weight is attached are used as the subjects forextracting the plurality of line segments.

Meanwhile, the subject to be used for extracting the plurality of linesegments is not particularly limited as long as the plurality of linesegments extending in the gravity direction can be extracted on thebasis of the subject captured in the image. For example, some structuressuch as buildings have an edge extending in the vertical direction (inother words, the gravity direction). Therefore, the intersection may bespecified by using an object known to have a portion extending in thevertical direction (gravity direction) therein as a subject, andextracting the portion (for example, an edge) extending in the verticaldirection from a captured image as the line segment.

For example, FIG. 7 is an explanatory diagram for describing an exampleof a calibration method according to Modification 1, and illustrates anexample of the case of using a structure such as building as a subjectfor extracting the plurality of line segments. Specifically, FIG. 7illustrates an example of an image of a case where a building having anedge extending in the vertical direction is captured to be looked downfrom obliquely above. In the example illustrated in FIG. 7, an object(building) with reference numeral M201 is set as a subject, and portionsrespectively corresponding to edges M203 a and M203 b extending in thevertical direction, of portions of the object, are extracted as the linesegments from the captured image. That is, an intersection of straightlines obtained by extending portions (line segments) respectivelycorresponding to the edges M203 a and M203 b in the image is specified,so that the gravity vector c in the local coordinate system of theimaging unit 131 can be calculated on the basis of the specificationresult of the intersection and the internal parameter of the imagingunit 131.

Furthermore, FIG. 8 is an explanatory diagram for describing anotherexample of a calibration method according to Modification 1, andillustrates another example of the case of using a structure such asbuilding as a subject for extracting the plurality of line segments.Specifically, FIG. 8 illustrates an example of an image of a case wherea building having an edge extending in the vertical direction iscaptured to be looked up from obliquely below. In the exampleillustrated in FIG. 8, an object (building) with reference numeral M211is set as a subject, and portions respectively corresponding to edgesM213 a and M213 b extending in the vertical direction, of portions ofthe object, are extracted as the line segments from a captured image.That is, an intersection of straight lines obtained by extendingportions (line segments) respectively corresponding to the edges M213 aand M213 b in the image is specified, so that the gravity vector c inthe local coordinate system of the imaging unit 131 can be calculated onthe basis of the specification result of the intersection and theinternal parameter of the imaging unit 131. Note that the intersectionspecified in the example in FIG. 8 corresponds to a vanishing point(infinite point) in a direction (vertically upward direction) oppositeto the gravity direction.

By performing the calibration by the method as described above, objectslocated in a surrounding area can be used for the calibration even if asubject is not prepared for the calibration, as in the example describedwith reference to FIGS. 3 to 5. Therefore, for example, in a case wherethe information acquisition device 130 is configured as a moving body,the information acquisition device 130 can perform the calibration inreal time while recognizing the environment around the informationacquisition device 130 by self-position estimation or the like.

Of course, the above-described example is a mere example, and does notlimit the calibration method according to the present embodiment. Forexample, a subject may not include a linear portion as long as thesubject captured in an image is analyzed and a plurality of linesegments extending in the gravity direction can be extracted from theimage. In other words, the subject may not include a linear portion aslong as information indicating the gravity direction corresponding tothe intersection can be extracted from the image. As a specific example,by setting an object held at a predetermined posture according to adirection in which the gravitational acceleration acts, as a subject, avector (in other words, a line segment) extending in the gravitydirection can be calculated in an image on the basis of the posture ofthe object captured in the image.

Furthermore, in the calibration according to the present embodiment, acommon subject may not be captured in images respectively correspondingto a plurality of viewpoints. That is, the subjects used for extractinga plurality of line segments may be different from each other among theplurality of viewpoints as long as the plurality of line segments (or aplurality of straight lines) extending in the gravity direction(vertical direction) can be extracted from the respective images for therespective viewpoints. As a more specific example, the image illustratedin FIG. 7 may be used as an image captured from a certain viewpoint, andthe image illustrated in FIG. 8 may be used as an image captured fromanother viewpoint different from the certain viewpoint. This alsosimilarly applies to the examples described with reference to FIGS. 3 to5.

As described above, as Modification 1, another example of thecalibration method according to the present embodiment has beendescribed focusing on a method of calculating the gravity directionvector c in the local coordinate system of the imaging unit 131 on thebasis of a subject in the image captured by the imaging unit 131.

<4.2. Modification 2: Example of Calculation Method of CoordinateConversion Function>

Next, as Modification 2, another example of the calibration methodaccording to the present embodiment will be described especiallyfocusing on a method of calculating the function R for performingconversion between the local coordinate system of the imaging unit 131and the local coordinate system of the detection unit 133.

Specifically, regarding the function R described as (Expression 5), theerrors (for example, the bias and scale factor) caused by the detectionunit 133 itself is assumed to be calibrated in advance. In contrast, inModification 2, an example of a method of estimating the bias and scalefactor of (that is, a method of calibrating) the detection unit 133 inaddition to the function R will be described.

First, a vector b representing the bias of the detection unit 133 isexpressed by (Expression 6) below in a case where biases in the xdirection, y direction, and z direction of the detection unit 133 in thelocal coordinate system of the detection unit 133 are b_(x), b_(y), andb_(z), respectively.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 6} \right\rbrack & \; \\{b = \begin{bmatrix}b_{x} \\b_{y} \\b_{z}\end{bmatrix}} & \left( {{Expression}\mspace{14mu} 6} \right)\end{matrix}$

Furthermore, a matrix S representing the scale factor of the detectionunit 133 is expressed by (Expression 7) below in a case where scalefactors in the x direction, y direction, and z direction of thedetection unit 133 in the local coordinate system of the detection unit133 are S_(x), S_(y), and S_(z), respectively.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 7} \right\rbrack & \; \\{S = \begin{bmatrix}s_{x} & 0 & 0 \\0 & s_{y} & 0 \\0 & 0 & s_{z}\end{bmatrix}} & \left( {{Expression}\mspace{14mu} 7} \right)\end{matrix}$

That is, the function R and the bias and scale factor of the detectionunit 133 can be estimated by considering the vector b indicating thebias described as (Expression 6) and the matrix S indicating the scalefactor described in (Expression 7) for the calculation expression of thefunction R described as (Expression 5). Specifically, the function R andthe bias and scale factor of the detection unit 133 can be calculated onthe basis of the calculation expression described as (Expression 8)below. g is the magnitude of the gravitational acceleration (9.8(m/s²)).

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 8} \right\rbrack & \; \\{R,b,{S = {\underset{R,b,S}{\arg\;\min}{\sum\limits_{n = {1\mspace{11mu}\ldots\mspace{11mu} N}}\left\{ {{g\frac{c_{n}}{c_{n}}} - {{RS}\left( {a_{n} - b} \right)}} \right\}}}}} & \left( {{Expression}\mspace{14mu} 8} \right)\end{matrix}$

As described above, as Modification 2, another example of thecalibration method according to the present embodiment has beendescribed especially focusing on a method of calculating the function Rfor performing conversion between the local coordinate system of theimaging unit 131 and the local coordinate system of the detection unit133.

<4.3. Modification 3: UI Example>

Next, as Modification 3, an example of a UI of the informationprocessing system according to the present embodiment will be described.In the present modification, an example of a UI for guiding the user toset a viewpoint at which the function R can be more stably calculatedwill be described.

Specifically, as described above with reference to FIG. 5, the viewpoint(in other words, the position and posture of the imaging unit 131, andthus the position and posture of the information acquisition device 130)is set to satisfy the condition described as (Expression 3), so that theintersection (and thus the vanishing point) can be more stablyspecified. Furthermore, it is favorable to set each viewpoint such thatthe position of the intersection specified from the image captured foreach viewpoint (that is, the position in the local coordinate system ofthe imaging unit 131) is different between at least two or moreviewpoints among the plurality of viewpoints.

Therefore, hereinafter, an example of a UI for guiding the user suchthat the information acquisition device 130 is held at the position andposture that satisfy the above conditions (that is, a more favorableviewpoint can be set) will be described. Note that, in the presentmodification, calibration is performed using the thread-like objectsM101 a and M101 b having fixed one ends and the other ends to which aweight is attached as subjects, as described with reference to FIGS. 3to 5, for convenience.

For example, FIG. 9 is an explanatory diagram for describing an outlineof the information processing system according to Modification 3, andillustrates an example of a UI for guiding the user to set a morefavorable viewpoint. Specifically, FIG. 9 illustrates an example of a UIfor guiding the user to set a viewpoint that satisfies the conditiondescribed as (Expression 3).

Specifically, in the example illustrated in FIG. 9, a guide V111 forguiding the position and posture of the information acquisition device130 to satisfy the condition described as (Expression 3) is superimposedon an image V110 (for example, a through image) captured by the imagingunit 131. In this case, the position and posture satisfy the conditiondescribed as (Expression 3) when the user adjusts the position andposture of the information acquisition device 130 such that thread-likeportions (that is, the portions extending in the gravity direction) ofthe objects M101 a and M101 b go along the guide V111.

Furthermore, in a case of setting each of a plurality of viewpoints,presentation of the guide V111 may be controlled such that the positionof a portion corresponding to the intersection of the guide V111 in theimage becomes different for each set viewpoint. Each of the plurality ofviewpoints is set to obtain a different result as the gravity directionvector c_(n) and the gravitational acceleration vector a_(n) among theplurality of viewpoints by the user being guided by such control.

Furthermore, FIG. 10 is an explanatory diagram for describing an outlineof the information processing system according to Modification 3, andillustrates another example of the UI for guiding the user to set a morefavorable viewpoint. Specifically, FIG. 10 illustrates an example of theUI for guiding the user to set a viewpoint in a more favorable mode, bypresenting a direction in which the position and posture of theinformation acquisition device 130 are changed to the user.

Specifically, in the example illustrated in FIG. 10, display informationV121 indicating the direction in which the position and posture of theinformation acquisition device 130 are changed is superimposed on animage V120 (for example, a through image) captured by the imaging unit131. In this case, the user changes the position and posture of theinformation acquisition device 130 along with the guidance of thedisplay information V121, so that the viewpoint is set in a morefavorable manner.

Note that the examples illustrated in FIGS. 9 and 10 are mere examples,and the UI for the guidance is not particularly limited as long as theUI can perform guidance such that the viewpoints are set in a morefavorable mode. Furthermore, an example of visually guiding the user bythe display information has been described with reference to FIGS. 9 and10. However, the guiding method is not necessarily limited. As aspecific example, the user may be guided by presentation of sound orforce sense.

As Modification 3, an example of the UI of the information processingsystem according to the present embodiment has been described. In thepresent modification, an example of the UI for guiding the user to set aviewpoint at which the function R can be more stably calculated has beendescribed.

5. HARDWARE CONFIGURATION

Next, an example of a hardware configuration of an informationprocessing device 900 configuring the information processing systemaccording to the present embodiment, as in the above-describedinformation processing device 100, will be described with reference toFIG. 11. FIG. 11 is a functional block diagram illustrating an exampleof a hardware configuration of an information processing device 900configuring an information processing system according to an embodimentof the present disclosure.

An information processing device 900 configuring the informationprocessing system 1 according to the present embodiment mainly includesa CPU 901, a ROM 902, and a RAM 903. Furthermore, the informationprocessing device 900 further includes a host bus 907, a bridge 909, anexternal bus 911, an interface 913, an input device 915, an outputdevice 917, a storage device 919, a drive 921, a connection port 923,and a communication device 925.

The CPU 901 functions as an arithmetic processing unit and a controldevice, and controls the entire operation or part thereof of theinformation processing device 900 according to various programs recordedin the ROM 902, the RAM 903, the storage device 919, or a removablerecording medium 927. The ROM 902 stores programs, arithmetic operationparameters, and the like used by the CPU 901. The RAM 903 primarilystores the programs used by the CPU 901, parameters that appropriatelychange in execution of the programs, and the like. The CPU 901, the ROM902, and the RAM 903 are mutually connected by the host bus 907configured by an internal bus such as a CPU bus. Note that the imageprocessing unit 101, the calculation unit 103, and the input/outputcontrol unit 105 described with reference to FIG. 2 can be implementedby the CPU 901, for example.

The host bus 907 is connected to the external bus 911 such as aperipheral component interconnect/interface (PCI) bus via the bridge909. Furthermore, the input device 915, the output device 917, thestorage device 919, the drive 921, the connection port 923, and thecommunication device 925 are connected to the external bus 911 via theinterface 913.

The input device 915 is an operation unit operated by the user, such asa mouse, a keyboard, a touch panel, a button, a switch, a lever, and apedal, for example. Furthermore, the input device 915 may be, forexample, a remote control unit (so-called remote controller) usinginfrared rays or other radio waves or an externally connected device 929such as a mobile phone or a PDA corresponding to an operation of theinformation processing device 900. Moreover, the input device 915 isconfigured by, for example, an input control circuit for generating aninput signal on the basis of information input by the user using theabove-described operation unit and outputting the input signal to theCPU 901, or the like. The user of the information processing device 900can input various data and give an instruction on processing operationsto the information processing device 900 by operating the input device915. Note that the input unit 153 described with reference to FIG. 2 canbe implemented by the input device 915, for example.

The output device 917 is configured by a device that can visually oraudibly notify the user of acquired information. Such devices includedisplay devices such as a CRT display device, a liquid crystal displaydevice, a plasma display device, an EL display device, a lamp, and thelike, sound output devices such as a speaker and a headphone, and aprinter device. The output device 917 outputs, for example, resultsobtained by various types of processing performed by the informationprocessing device 900. Specifically, the display device displays theresults of the various types of processing performed by the informationprocessing device 900 as texts or images. Meanwhile, the sound outputdevice converts an audio signal including reproduced sound data, voicedata, or the like into an analog signal and outputs the analog signal.Note that the output unit 155 described with reference to FIG. 2 can beimplemented by the output device 917, for example.

The storage device 919 is a device for data storage configured as anexample of a storage unit of the information processing device 900. Thestorage device 919 is configured by a magnetic storage device such as ahard disk drive (HDD), a semiconductor storage device, an opticalstorage device, a magneto-optical storage device, or the like, forexample. The storage device 919 stores programs executed by the CPU 901,various data, and the like.

The drive 921 is a reader/writer for a recording medium, and is built inor is externally attached to the information processing device 900. Thedrive 921 reads out information recorded on the removable recordingmedium 927 such as a mounted magnetic disk, optical disk,magneto-optical disk, or semiconductor memory, and outputs theinformation to the RAM 903. Furthermore, the drive 921 can also write arecord on the removable recording medium 927 such as the mountedmagnetic disk, optical disk, magneto-optical disk, or semiconductormemory. The removable recording medium 927 is, for example, a DVDmedium, an HD-DVD medium, a Blu-ray (registered trademark) medium, orthe like. Furthermore, the removable recording medium 927 may be acompact flash (CF (registered trademark)), a flash memory, a securedigital (SD) memory card, or the like. Furthermore, the removablerecording medium 927 may be, for example, an integrated circuit (IC)card on which a non-contact IC chip is mounted, an electronic device, orthe like. Note that the storage unit 151 described with reference toFIG. 2 can be implemented by at least either the RAM 903 or the storagedevice 919, for example.

The connection port 923 is a port for being directly connected to theinformation processing device 900. Examples of the connection port 923include a universal serial bus (USB) port, an IEEE 1394 port, a smallcomputer system interface (SCSI) port, and the like. Other examples ofthe connection port 923 include an RS-232C port, an optical audioterminal, a high-definition multimedia interface (HDMI) (registeredtrademark) port, and the like. By connecting the externally connecteddevice 929 to the connection port 923, the information processing device900 directly acquires various data from the externally connected device929 and provides various data to the externally connected device 929.

The communication device 925 is, for example, a communication interfaceconfigured by a communication device for being connected to acommunication network (network) 931, and the like The communicationdevice 925 is, for example, a communication card for a wired or wirelesslocal area network (LAN), Bluetooth (registered trademark), a wirelessUSB (WUSB), or the like. Furthermore, the communication device 925 maybe a router for optical communication, a router for an asymmetricdigital subscriber line (ADSL), a modem for various communications, orthe like. The communication device 925 can transmit and receive signalsand the like, for example, to and from the Internet and othercommunication devices in accordance with a predetermined protocol suchas TCP/IP, for example. Furthermore, a communication network 931connected to the communication device 925 is configured by a network orthe like connected by wire or wirelessly, and may be, for example, theInternet, home LAN, infrared communication, radio wave communication,satellite communication, or the like.

An example of the hardware configuration which can implement thefunctions of the information processing device 900 configuring theinformation processing system 1 according to the embodiment of thepresent disclosure has been described. Each of the above-describedconfiguration elements may be configured using general-purpose membersor may be configured by hardware specialized for the function of eachconfiguration element. Therefore, the hardware configuration to be usedcan be changed as appropriate according to the technical level of thetime of carrying out the present embodiment. Note that, although notillustrated in FIG. 11, various configurations corresponding to theinformation processing device 900 configuring the information processingsystem 1 according to the present embodiment are naturally provided.

Note that a computer program for realizing the functions of theinformation processing device 900 configuring the information processingsystem 1 according to the above-described present embodiment can beprepared and implemented on a personal computer or the like.Furthermore, a computer-readable recording medium in which such acomputer program is stored can be provided. The recording medium is, forexample, a magnetic disk, an optical disk, a magneto-optical disk, aflash memory, or the like. Furthermore, the above computer program maybe delivered via, for example, a network without using a recordingmedium. Furthermore, the number of computers that execute the computerprogram is not particularly limited. For example, a plurality ofcomputers (for example, a plurality of servers or the like) may executethe computer program in cooperation with one another. Note that a singlecomputer or a plurality of computers cooperating with one another isalso referred to as a “computer system”.

6. APPLICATION

Next, in the information processing system according to the embodimentof the present disclosure, an example of the information acquisitiondevice 130 to be calibrated will be described.

The technology regarding calibration by the information processingdevice 100 according to the present embodiment can be applied to adevice in which the imaging unit 131 and the detection unit 133 areintegrally supported. At this time, the imaging unit 131 and thedetection unit 133 may be configured such that one unit is attachable toand detachable from the other unit. That is, the technology regardingcalibration can be applied to a device configured such that one unit isheld with respect to the other unit so as to temporarily fix at leastthe relative posture relationship between the imaging unit 131 and thedetection unit 133 (for example, fix the relative posture relationshipat the time of acquiring the information for calibration). Examples ofsuch a device include a device that is configured to be portable, adevice assumed to be carried by a user during use, and the like. Morespecific examples include a communication terminal such as a smartphone,and a wearable device used by being worn on a part of the user's body.Furthermore, in recent years, some controllers and the like used by theuser who holds a housing thereof are provided with configurationscorresponding to the imaging unit 131 and the detection unit 133 inorder to implement gesture input and the like. The technology regardingcalibration can also be applied to such controllers. Furthermore, otherexamples of the device to which the technology regarding calibrationinclude a movable body such as a vehicle or a drone, and a movablyconfigured device such as a so-called autonomous robot.

Here, as an example of the device to which the technology regardingcalibration is applicable, an example of a specific configuration of aso-called head-mounted device such as a head mounted display (HMD) usedfor implementing augmented reality (AR) or virtual reality (VR) will bedescribed. For example, FIG. 12 is an explanatory diagram for describingan application of an embodiment of the present disclosure, andillustrates an example of a configuration of an input/output device 20that can be calibrated by the information processing device 100.

The input/output device 20 is configured as a so-called head-mounteddevice worn on at least part of the head of the user and used by theuser. For example, in the example illustrated in FIG. 12, theinput/output device 20 is configured as a so-called eyewear-type(glasses-type) device, and at least either a lens 293 a or a lens 293 bis configured as a transmission-type display (display unit 211).Furthermore, the input/output device 20 includes first imaging units 201a and 201 b, second imaging units 203 a and 203 b, an operation unit207, and a holding unit 291 corresponding to a frame of the glasses. Theholding unit 291 holds the display unit 211, the first imaging units 201a and 201 b, the second imaging units 203 a and 203 b, and the operationunit 207 to have a predetermined positional relationship with respect tothe head of the user when the input/output device 20 is mounted on thehead of the user. Furthermore, although not illustrated in FIG. 12, theinput/output device 20 may be provided with a sound collection unit forcollecting a voice of the user.

Here, a more specific configuration of the input/output device 20 willbe described. For example, in the example illustrated in FIG. 12, thelens 293 a corresponds to a lens on a right eye side, and the lens 293 bcorresponds to a lens on a left eye side. In other words, the holdingunit 291 holds the display unit 211 such that the display unit 211 (inother words, the lenses 293 a and 293 b) is located in front of the eyesof the user in the case where the input/output device 20 is mounted.

The first imaging units 201 a and 201 b are configured as so-calledstereo cameras and are held by the holding unit 291 to face a directionin which the head of the user faces (in other words, the front of theuser) when the input/output device 20 is mounted on the head of theuser. At this time, the first imaging unit 201 a is held near the user'sright eye, and the first imaging unit 201 b is held near the user's lefteye. The first imaging units 201 a and 201 b capture a subject locatedin front of the input/output device 20 (in other words, a real objectlocated in a real space) from different positions on the basis of such aconfiguration. Thereby, the input/output device 20 acquires images ofthe subject located in front of the user and can calculate a distance tothe subject from the input/output device 20 on the basis of a parallaxbetween the images respectively captured by the first imaging units 201a and 201 b.

Note that the configuration and method are not particularly limited aslong as the distance between the input/output device 20 and the subjectcan be measured. As a specific example, the distance between theinput/output device 20 and the subject may be measured on the basis of amethod such as multi-camera stereo, moving parallax, time of flight(TOF), or structured light. Here, the TOF is a method of obtaining animage (so-called distance image) including a distance (depth) to asubject on the basis of a measurement result by projecting light such asinfrared light on the subject and measuring a time required for theprojected light to be reflected by the subject and return, for eachpixel. Furthermore, the structured light is a method of obtaining adistance image including a distance (depth) to a subject on the basis ofchange in a pattern obtained from a capture result by irradiating thesubject with the pattern of light such as infrared light and capturingthe pattern. Furthermore, the moving parallax is a method of measuring adistance to a subject on the basis of a parallax even in a so-calledmonocular camera. Specifically, the subject is captured from differentviewpoints from each other by moving the camera, and the distance to thesubject is measured on the basis of the parallax between the capturedimages. Note that, at this time, the distance to be subject can bemeasured with more accuracy by recognizing a moving distance and amoving direction of the camera using various sensors. Note that theconfiguration of the imaging unit (for example, the monocular camera,the stereo camera, or the like) may be changed according to the distancemeasuring method.

Furthermore, the second imaging units 203 a and 203 b are held by theholding unit 291 such that eyeballs of the user are located withinrespective imaging ranges when the input/output device 20 is mounted onthe head of the user. As a specific example, the second imaging unit 203a is held such that the user's right eye is located within the imagingrange. The direction in which the line-of-sight of the right eye isdirected can be recognized on the basis of an image of the eyeball ofthe right eye captured by the second imaging unit 203 a and a positionalrelationship between the second imaging unit 203 a and the right eye, onthe basis of such a configuration. Similarly, the second imaging unit203 b is held such that the user's left eye is located within theimaging range. In other words, the direction in which the line-of-sightof the left eye is directed can be recognized on the basis of an imageof the eyeball of the left eye captured by the second imaging unit 203 band a positional relationship between the second imaging unit 203 b andthe left eye. Note that the example in FIG. 12 illustrates theconfiguration in which the input/output device 20 includes both thesecond imaging units 203 a and 203 b. However, only one of the secondimaging units 203 a and 203 b may be provided.

The operation unit 207 is configured to receive an operation on theinput/output device 20 from the user. The operation unit 207 may beconfigured by, for example, an input device such as a touch panel or abutton. The operation unit 207 is held at a predetermined position ofthe input/output device 20 by the holding unit 291. For example, in theexample illustrated in FIG. 12, the operation unit 207 is held at aposition corresponding to a temple of the glasses.

Furthermore, the input/output device 20 may be provided with, forexample, an acceleration sensor and an angular velocity sensor (gyrosensor) and configured to be able to detect a motion of the head (inother words, a posture of the input/output device 20 itself) of the userwearing the input/output device 20. As a specific example, theinput/output device 20 may detect components in a yaw direction, a pitchdirection, and a roll direction as the motion of the head of the user,thereby recognizing a change in at least either the position or postureof the head of the user.

The input/output device 20 may recognize changes in its own position andposture in the real space according to the motion of the head of theuser on the basis of the above configuration, for example. Furthermore,at this time, the input/output device 20 may present the virtual content(in other words, the virtual object) on the display unit 211 tosuperimpose the virtual content on the real object located in the realspace on the basis of the so-called AR technology. To implement suchcontrol, the above-described technology according to the presentembodiment (that is, the technology regarding calibration) can beapplied to, for example, calibration for a relative relationship betweenpostures of the first imaging units 201 a and 201 b and postures of theacceleration sensor and the angular velocity sensor described above.

Note that examples of a head mounted display (HMD) device applicable asthe input/output device 20 include a see-through HMD, a videosee-through HMD, and a retinal projection HMD, in the case of assumingapplication of the AR technology.

The see-through HMD uses, for example, a half mirror or a transparentlight guide plate to hold a virtual image optical system including atransparent light guide or the like in front of the eyes of the user,and displays an image inside the virtual image optical system.Therefore, the user wearing the see-through HMD can take the externalscenery into view while viewing the image displayed inside the virtualimage optical system. With such a configuration, the see-through HMD cansuperimpose an image of the virtual object on an optical image of thereal object located in the real space according to the recognitionresult of at least one of the position or posture of the see-through HMDon the basis of the AR technology, for example. Note that a specificexample of the see-through HMD includes a so-called glasses-typewearable device in which a portion corresponding to a lens of glasses isconfigured as a virtual image optical system. For example, theinput/output device 20 illustrated in FIG. 12 corresponds to an exampleof the see-through HMD.

In a case where the video see-through HMD is mounted on the head or faceof the user, the video see-through HMD is mounted to cover the eyes ofthe user, and a display unit such as a display is held in front of theeyes of the user. Furthermore, the video see-through HMD includes animaging unit for capturing surrounding scenery, and causes the displayunit to display an image of the scenery in front of the user captured bythe imaging unit. With such a configuration, the user wearing the videosee-through HMD has a difficulty in directly taking the external sceneryinto view but the user can confirm the external scenery with the imagedisplayed on the display unit. Furthermore, at this time, the videosee-through HMD may superimpose the virtual object on an image of theexternal scenery according to the recognition result of at least one ofthe position or posture of the video see-through HMD on the basis of theAR technology, for example.

The retinal projection HMD has a projection unit held in front of theeyes of the user, and an image is projected from the projection unittoward the eyes of the user such that the image is superimposed on theexternal scenery. More specifically, in the retinal projection HMD, animage is directly projected from the projection unit onto the retinas ofthe eyes of the user, and the image is imaged on the retinas. With sucha configuration, the user can view a clearer image even in a case wherethe user has myopia or hyperopia. Furthermore, the user wearing theretinal projection HMD can take the external scenery into view evenwhile viewing the image projected from the projection unit. With such aconfiguration, the retinal projection HMD can superimpose an image ofthe virtual object on an optical image of the real object located in thereal space according to the recognition result of at least one of theposition or posture of the retinal projection HMD on the basis of the ARtechnology, for example.

Furthermore, an HMD called an immersive HMD can be applied assumingapplication of the VR technology, in addition to the above-describedexamples. The immersive HMD is mounted to cover the eyes of the user,and a display unit such as a display is held in front of the eyes of theuser, similarly to the video see-through HMD. Therefore, the userwearing the immersive HMD has a difficulty in directly taking anexternal scenery (in other words, scenery of a real world) into view,and only an image displayed on the display unit comes into view. Withsuch a configuration, the immersive HMD can provide an immersive feelingto the user who is viewing the image.

The examples of the configurations of the head-mounted device used toimplement AR or VR have been described with reference to FIG. 12, asexamples of the device to which the technology regarding calibration isapplicable.

7. CONCLUSION

As described above, the information processing device according to anembodiment of the present disclosure acquires an image captured by animaging unit and a detection result of gravitational acceleration by adetection unit supported by the imaging unit, for each of a plurality ofviewpoints different from each other. Furthermore, the informationprocessing device extracts a plurality of line segments extending in thegravity direction from the image and specifies an intersection ofstraight lines obtained by respectively extending the plurality of linesegments, on the basis of a subject captured in the image of the eachviewpoint. Then, the information processing device calculates a relativeposture relationship between the imaging unit and the detection unit onthe basis of the detection result of the gravitational accelerationacquired for each of the plurality of viewpoints and the intersectionspecified for the each of the plurality of viewpoints. As a specificexample, the information processing device calculates the function R forconverting the posture in one coordinate system into the posture in theother coordinate system between the first coordinate system associatedwith the imaging unit and the second coordinate system associated withthe detection unit.

With the above configuration, the correspondence (calibration) betweenthe posture of the imaging unit and the posture of the detection unit,to which the errors related to attachment of the imaging unit and thedetection unit are added, can be implemented with a smaller calculationamount. Furthermore, according to the technology of the presentdisclosure, the above correspondence can be implemented by simplefacility and simple procedures without requiring special facility andcomplicated procedures. As described above, according to the informationprocessing device of the embodiment of the present disclosure, thecorrespondence between the posture of the imaging unit and the postureof the detection unit (in other words, the coordinate conversion betweenthe imaging unit and the detection unit) can be implemented in a morefavorable mode. Thereby, the recognition technology using the analysisresult of the captured image by the imaging unit and the detectionresult by the detection unit can be implemented in a more favorablemode. Therefore, for example, a more accurate viewpoint change withsmall superimposition deviation can be reproduced in implementing AR orVR.

Although the favorable embodiment of the present disclosure has beendescribed in detail with reference to the accompanying drawings, thetechnical scope of the present disclosure is not limited to suchexamples. It is obvious that persons having ordinary knowledge in thetechnical field of the present disclosure can conceive various changesand alterations within the scope of the technical idea described in theclaims, and it is naturally understood that these changes andalterations belong to the technical scope of the present disclosure.

Furthermore, the effects described in the present specification aremerely illustrative or exemplary and are not restrictive. That is, thetechnology according to the present disclosure can exhibit other effectsobvious to those skilled in the art from the description of the presentspecification together with or in place of the above-described effects.

Note that following configurations also belong to the technical scope ofthe present disclosure.

(1) An information processing device including:

an acquisition unit configured to acquire an image captured by animaging unit and a detection result of a gravitational acceleration by adetection unit supported by the imaging unit, for each of a plurality ofviewpoints different from one another;

an image processing unit configured to extract a plurality of linesegments extending in a gravity direction from the image and specify anintersection of straight lines obtained by respectively extending theplurality of line segments on the basis of a subject captured in theimage for the each viewpoint; and

a calculation unit configured to calculate a relative posturerelationship between the imaging unit and the detection unit on thebasis of the detection result of the gravitational acceleration acquiredfor each of the plurality of viewpoints and the intersection specifiedfor each of the plurality of viewpoints.

(2) The information processing device according to (1), in which thecalculation unit makes a first coordinate system associated with theimaging unit and a second coordinate system associated with thedetection unit correspond to each other on the basis of the detectionresult of the gravitational acceleration acquired for the each viewpointand a specification result of the intersection for the each viewpoint.

(3) The information processing device according to (2), in which

the calculation unit

calculates a vector in the gravity direction in the first coordinatesystem at the viewpoint on the basis of the specification result of theintersection for the each viewpoint, and

makes the first coordinate system and the second coordinate systemcorrespond to each other on the basis of the detection result of thegravitational acceleration acquired for each of the plurality ofviewpoints and the vector in the gravity direction calculated for theeach of the plurality of viewpoints.

(4) The information processing device according to (3), in which thecalculation unit calculates a function for converting a posture in onecoordinate system into a posture in another coordinate system betweenthe first coordinate system and the second coordinate system on thebasis of the vector in the gravity direction in the first coordinatesystem and a vector according to the gravitational acceleration in thesecond coordinate system.

(5) The information processing device according to any one of (1) to(4), in which the image processing unit extracts at least part of theplurality of line segments on the basis of a posture of an object heldat the posture according to gravity, the object being captured as thesubject in the image for the each viewpoint.

(6) The information processing device according to (5), in which theimage processing unit extracts the line segment on the basis of theposture of the object, for each of a plurality of the objects capturedas the subject in the image for the each viewpoint.

(7) The information processing device according to (6), in which atleast part of the plurality of objects is a thread-like object with afixed one end.

(8) The information processing device according to any one of (1) to(4), in which the image processing unit extracts at least part of theplurality of line segments on the basis of an edge extending in thegravity direction of an object captured as the subject in the image forthe each viewpoint.

(9) The information processing device according to (8), in which theobject is a building having the edge extending in the gravity directionin at least part of the building.

(10) The information processing device according to any one of (1) to(9), in which at least two or more viewpoints included in the pluralityof viewpoints are set such that positions of the intersections in thecoordinate system associated with the imaging unit are different fromeach other, the intersections being specified from the images capturedby the imaging unit from the viewpoints.

(11) The information processing device according to any one of (1) to(10), further including: an output control unit configured to cause anoutput unit to output information for guiding movement of a position ofthe viewpoint such that, in at least two or more viewpoints of theplurality of viewpoints, positions of the intersections in thecoordinate system associated with the imaging unit are different fromeach other, the intersections being specified from the images capturedby the imaging unit from the viewpoints.

(12) An information processing method, by a computer, including:

acquiring an image captured by an imaging unit and a detection result ofa gravitational acceleration by a detection unit supported by theimaging unit, for each of a plurality of viewpoints different from oneanother;

extracting a plurality of line segments extending in a gravity directionfrom the image and specify an intersection of straight lines obtained byrespectively extending the plurality of line segments on the basis of asubject captured in the image for the each viewpoint; and

calculating a relative posture relationship between the imaging unit andthe detection unit on the basis of the detection result of thegravitational acceleration acquired for each of the plurality ofviewpoints and the intersection specified for each of the plurality ofviewpoints.

(13) A program for causing a computer to execute:

acquiring an image captured by an imaging unit and a detection result ofa gravitational acceleration by a detection unit supported by theimaging unit, for each of a plurality of viewpoints different from oneanother;

extracting a plurality of line segments extending in a gravity directionfrom the image and specify an intersection of straight lines obtained byrespectively extending the plurality of line segments on the basis of asubject captured in the image for the each viewpoint; and

calculating a relative posture relationship between the imaging unit andthe detection unit on the basis of the detection result of thegravitational acceleration acquired for each of the plurality ofviewpoints and the intersection specified for each of the plurality ofviewpoints.

REFERENCE SIGNS LIST

-   -   1 Information processing system    -   100 Information processing device    -   101 Image processing unit    -   103 Calculation unit    -   105 Input/output control unit    -   130 Information acquisition device    -   131 Imaging unit    -   133 Detection unit    -   151 Storage unit    -   153 Input unit    -   155 Output unit

The invention claimed is:
 1. An information processing devicecomprising: an acquisition unit configured to acquire an image capturedby an imaging unit and a detection result of a gravitationalacceleration by a detection unit supported by the imaging unit, for eachof a plurality of viewpoints different from one another; an imageprocessing unit configured to extract a plurality of line segmentsextending in a gravity direction from the image and specify anintersection of straight lines obtained by respectively extending theplurality of line segments on a basis of a subject captured in the imagefor the each viewpoint; a calculation unit configured to calculate arelative posture relationship between the imaging unit and the detectionunit on a basis of the detection result of the gravitationalacceleration acquired for each of the plurality of viewpoints and theintersection specified for the each of the plurality of viewpoints; andan output control unit configured to cause an output unit to outputinformation for guiding movement of a position of the viewpoint suchthat positions of the intersections in a coordinate system associatedwith the imaging unit are different from each other, wherein theacquisition unit, the image processing unit, the calculation unit, andthe output control unit are each implemented via at least one processor.2. The information processing device according to claim 1, wherein thecalculation unit is further configured to make a first coordinate systemassociated with the imaging unit and a second coordinate systemassociated with the detection unit correspond to each other on a basisof the detection result of the gravitational acceleration acquired forthe each viewpoint and a specification result of the intersection forthe each viewpoint.
 3. The information processing device according toclaim 2, wherein the calculation unit is further configured to calculatea vector in the gravity direction in the first coordinate system at theviewpoint on a basis of the specification result of the intersection forthe each viewpoint, and make the first coordinate system and the secondcoordinate system correspond to each other on a basis of the detectionresult of the gravitational acceleration acquired for each of theplurality of viewpoints and the vector in the gravity directioncalculated for the each of the plurality of viewpoints.
 4. Theinformation processing device according to claim 3, wherein thecalculation unit is further configured to calculate a function forconverting a posture in one coordinate system into a posture in anothercoordinate system between the first coordinate system and the secondcoordinate system on a basis of the vector in the gravity direction inthe first coordinate system and a vector according to the gravitationalacceleration in the second coordinate system.
 5. The informationprocessing device according to claim 1, wherein the image processingunit is further configured to extract at least part of the plurality ofline segments on a basis of a posture of an object held at the postureaccording to gravity, the object being captured as the subject in theimage for the each viewpoint.
 6. The information processing deviceaccording to claim 5, wherein the image processing unit is furtherconfigured to extract the line segment on a basis of the posture of theobject, for each of a plurality of the objects captured as the subjectin the image for the each viewpoint.
 7. The information processingdevice according to claim 6, wherein at least part of the plurality ofobjects is a thread-like object with a fixed one end.
 8. The informationprocessing device according to claim 1, wherein the image processingunit is further configured to extract at least part of the plurality ofline segments on a basis of an edge extending in the gravity directionof an object captured as the subject in the image for the eachviewpoint.
 9. The information processing device according to claim 8,wherein the object is a building having the edge extending in thegravity direction in at least part of the building.
 10. The informationprocessing device according to claim 1, wherein at least two or moreviewpoints included in the plurality of viewpoints are set such thatpositions of the intersections in the coordinate system associated withthe imaging unit are different from each other, the intersections beingspecified from the images captured by the imaging unit from theviewpoints.
 11. The information processing device according to claim 1,wherein the output control unit is further configured to cause theoutput unit to output the information for guiding the movement of theposition of the viewpoint such that, in at least two or more viewpointsof the plurality of viewpoints, the positions of the intersections inthe coordinate system associated with the imaging unit are differentfrom each other, the intersections being specified from the imagescaptured by the imaging unit from the viewpoints.
 12. An informationprocessing method, by a computer, comprising: acquiring an imagecaptured by an imaging unit and a detection result of a gravitationalacceleration by a detection unit supported by the imaging unit, for eachof a plurality of viewpoints different from one another; extracting aplurality of line segments extending in a gravity direction from theimage and specify an intersection of straight lines obtained byrespectively extending the plurality of line segments on a basis of asubject captured in the image for the each viewpoint; calculating arelative posture relationship between the imaging unit and the detectionunit on a basis of the detection result of the gravitationalacceleration acquired for each of the plurality of viewpoints and theintersection specified for the each of the plurality of viewpoints; andoutputting information for guiding movement of a position of theviewpoint such that positions of the intersections in a coordinatesystem associated with the imaging unit are different from each other.13. A non-transitory computer-readable medium having embodied thereon aprogram, which when executed by a computer causes the computer toexecute an information processing method, the method comprising:acquiring an image captured by an imaging unit and a detection result ofa gravitational acceleration by a detection unit supported by theimaging unit, for each of a plurality of viewpoints different from oneanother; extracting a plurality of line segments extending in a gravitydirection from the image and specify an intersection of straight linesobtained by respectively extending the plurality of line segments on abasis of a subject captured in the image for the each viewpoint;calculating a relative posture relationship between the imaging unit andthe detection unit on a basis of the detection result of thegravitational acceleration acquired for each of the plurality ofviewpoints and the intersection specified for the each of the pluralityof viewpoints; and outputting information for guiding movement of aposition of the viewpoint such that positions of the intersections in acoordinate system associated with the imaging unit are different fromeach other.